Adds enhanced search #287

slowe · 2025-06-02T14:13:40Z

This adds a first draft of an enhanced search interface that uses ReadTheDocs's API to search across multiple projects as requested by @Cadair.

This uses the existing search feature but then adapts the interface to add inline results rather than redirect to the search results page. If CORS is not enabled (with ReadTheDocs) it will redirect to the standard site search page. Alternatively, you can add ?debug to the URL to use https://corsproxy.io/ to get around CORS issues.

The inline search results will be arranged into tabs showing "All" results and results broken down by project. By default, the script will attempt to find projects from the page. It will look for elements with class="nav-link" from a menu item with id="Documentation". However, you can manually over-ride this behaviour by using the Javascript variable set_search_config e.g.

<script>
const set_search_config = {
  "no-results":{
   "label":"Nothing to show here."
  },
  "load-more":{
    "label": "Load more results",
    "class": "btn btn-lg btn-sunpy btn-sunpy1"
  },
  "projects":[
    {"name":"sunpy","link":"https://docs.sunpy.org/"},
    {"name":"ndcube","link":"https://docs.sunpy.org/projects/ndcube/"},
    {"name":"sunraster","link":"https://docs.sunpy.org/projects/sunraster/"},
    {"name":"aiapy","link":"https://aiapy.readthedocs.io/"}
  ]
};
</script>
<script src="path/to/sunpy_enhanced_search.js"></script>

where:

no-results - an optional object to set the label that gets displayed when there are no results;
load-more - an optional object to set the class/label of the "load more" button;
projects - an ordered array of projects to include (if empty this will be constructed from the ".nav-link" items found within #Documentation). The order of this array sets the order of the search results tabs.

Cadair · 2025-06-02T15:39:09Z

src/sunpy_sphinx_theme/theme/sunpy/static/sunpy_enhanced_search.js

+			let path = (page || ("api/v3/search/?q=" + projstr + "+" + encodeURIComponent(str)));
+			if(debug) {
+				url = "https://corsproxy.io/?https://readthedocs.org/" + path;
+			}else{
+				url = "/_/" + path;
+			};


@slowe I did bad things to try and get the RTD preview working.

and I broke "load more" in the process 😭

I think I fixed it for load more, but I made it much more ugly in the process!

Cadair · 2025-06-02T16:01:52Z

@slowe here's some things I noticed while testing with the RTD preview:

Using the keyboard I appear to only be able to move between the first two tabs (pressing left and right)
I'm getting the Load More button when I don't really expect it to appear, also using the keyboard right shortcut seems to cause it to pop up (maybe related to above issue?) - oh it seems to always show? Is there any way to only show it if there are hidden results?
The styling on the load more button could be nicer.

slowe · 2025-06-13T09:47:59Z

First thing to note: I'm only able to test on a local file system so I may not have the same situation as you.

Using the keyboard I appear to only be able to move between the first two tabs (pressing left and right)

This is probably down to the change to hide empty tabs. I'll have a look.

I'm getting the Load More button when I don't really expect it to appear

Can you give an example of when? I am not seeing the same thing so it is probably down to a difference in expectation.

Is there any way to only show it if there are hidden results?

By "hidden" do you mean "not loaded"?

The styling on the load more button could be nicer.

Please can you advise on what that should be?

slowe · 2025-06-13T10:10:41Z

To clarify what I think is happening around the "Load more" button... I make one request to the ReadTheDocs API that includes all the projects. ReadTheDocs then returns up to 50 results at a time e.g. https://readthedocs.org/api/v3/search/?q=project:sunpy+project:ndcube+project:sunraster+project:aiapy+fits

I'm then doing the totalling up by project. It is only possible to know the count by project for the results that have been loaded so far. So there may only be, say, "4" results loaded in a particular project but that doesn't mean there couldn't be more in the next batch of 50 results. The "Load more" button should keep appearing until all the results have been loaded.

An alternative would be to make n requests to ReadTheDocs, one for each project every time the typeahead activates. This is inefficient and, potentially, acts more like a denial-of-service attack. Is it worth the number of requests to remove a slight, potential, confusion? I would also hope that most people have found what they were looking for within the first 50.

slowe · 2025-06-13T10:17:06Z

The styling on the "Load more" button is the same as the styling on the buttons on the front of sunpy.org as it uses the classes load-more btn btn-lg btn-sunpy btn-sunpy1. So it should inherit the style from the theme. The load-more class should allow you to specifically style that button in your theme style (the JS code doesn't apply any styling of its own to it).

slowe · 2025-06-13T10:28:34Z

Ah ha. This is how it looks for me locally:

And this is on the "website preview" link (set to debug so that results load) https://output.circle-artifacts.com/output/job/00b5f97b-26f5-4665-9602-9ff4a13b7b61/artifacts/0/.tmp/website/sunpy.org/_build/html/index.html?debug=true

But what I said about the styles is because the classes are set in set_search_config i.e.

const set_search_config = {
  "no-results":{
   "label":"Nothing to show here."
  },
  "load-more":{
    "label": "Load more results",
    "class": "btn btn-lg btn-sunpy btn-sunpy1"
  },
  "projects":[
    {"name":"sunpy","link":"https://docs.sunpy.org/"},
    {"name":"ndcube","link":"https://docs.sunpy.org/projects/ndcube/"},
    {"name":"sunraster","link":"https://docs.sunpy.org/projects/sunraster/"},
    {"name":"aiapy","link":"https://aiapy.readthedocs.io/"}
  ]
};

I didn't add btn btn-lg btn-sunpy btn-sunpy1 by default (only load-more) because I wasn't clear if they were part of the overall theme or project-specific.

Cadair · 2025-06-18T06:29:52Z

It is only possible to know the count by project for the results that have been loaded so far. So there may only be, say, "4" results loaded in a particular project but that doesn't mean there couldn't be more in the next batch of 50 results. The "Load more" button should keep appearing until all the results have been loaded.

I would also hope that most people have found what they were looking for within the first 50.

Ah ok, that makes sense. It just surprised me that I could see a (4) for a project followed by load more. I assume that if there are no more pages of results to load for any project the load more wont show?

I didn't add btn btn-lg btn-sunpy btn-sunpy1 by default (only load-more) because I wasn't clear if they were part of the overall theme or project-specific.

Ah I see, I think we can safely add them by default they are defined in this theme. (Which I see you have done.)

@slowe Could you have a look over the changes I made to the cors proxy code if you haven't already?

I think there's a couple of things left to do before we merge this, but I think they are mostly on me:

Add python / sphinx side config to set projects and general config from the html_theme_config variable.
Figure out the CSS theming on the button which isn't working on RTD but is working locally.

A more compact version of the URL building. Remove/define some variables.

Switching to classes `sd-btn sd-bg-primary sd-bg-text-primary` from https://sunpy--287.org.readthedocs.build/projects/sunpy-sphinx-theme/287/web-components.html

slowe · 2025-06-18T09:12:33Z

I've changed the default "Load more" button styles to load-more btn sd-btn sd-bg-primary sd-bg-text-primary. The sd- prefixed styles are taken from https://sunpy--287.org.readthedocs.build/projects/sunpy-sphinx-theme/287/web-components.html

I assume that if there are no more pages of results to load for any project the load more wont show?

That's the case, yes. It basically depends on the existence (or not) of a next page in the ReadTheDocs API result.

Would it help if the icon in each project tab was a filter icon (e.g. https://icons.getbootstrap.com/icons/funnel-fill/) to visually distinguish those tabs from the "All" tab?

Cadair · 2025-06-18T09:28:39Z

Would it help if the icon in each project tab was a filter icon

Yeah, I think that's a good idea.

slowe · 2025-06-24T08:07:30Z

@Cadair How about this? https://output.circle-artifacts.com/output/job/b4dcff59-b622-479c-8eeb-641acf71eda0/artifacts/0/.tmp/website/sunpy.org/_build/html/index.html?debug=true

Cadair · 2025-09-02T14:53:21Z

pre-commit.ci autofix

for more information, see https://pre-commit.ci

Cadair · 2025-09-02T15:01:36Z

~~The CSS is busted on the dark theme.~~

I fixed it.

nabobalis · 2025-09-02T21:25:55Z

src/sunpy_sphinx_theme/theme/sunpy/theme.conf

+# Enable project-wide search with RTD
+rtd_search = True
+rtd_search_projects =
+rtd_search_load_more_label =
+rtd_search_no_results_label =


Are these explained anywhere?

Copilot

Pull Request Overview

This PR adds enhanced search functionality using ReadTheDocs API to enable cross-project search with inline results. The implementation provides a rich interface with tabbed results, keyboard navigation, and fallback to standard search when CORS is not enabled.

Key changes:

New RTD-powered search interface with inline results and tabbed project filtering
JavaScript and CSS files to implement the enhanced search experience
Theme configuration options for customizing search behavior
Integration with existing navbar links to automatically detect searchable projects

Reviewed Changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
theme.conf	Adds configuration options for RTD search functionality
rtd_enhanced_search.js	Core JavaScript implementation for enhanced search interface
rtd_enhanced_search.css	Styling for the enhanced search modal and results
cards.py	Updates CSS file path reference
init.py	Integrates enhanced search into theme setup and configuration

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-09-02T22:52:06Z

src/sunpy_sphinx_theme/__init__.py

+                    raise ValueError(err)
+            return out_links
+
+        projects = filter_doc_links(doc_links)


The variable projects is used but only defined inside the conditional block when search_projects is None. If search_projects is not None, projects will be undefined, causing a NameError when used in the JSON serialization on line 194.

Suggested change

projects = filter_doc_links(doc_links)

projects = filter_doc_links(doc_links)

else:

projects = search_projects

Copilot · 2025-09-02T22:52:07Z

src/sunpy_sphinx_theme/theme/sunpy/static/js/rtd_enhanced_search.js

+    function highlightAllMatches(text, search) {
+      const regex = new RegExp(`(${search})`, "gi");
+      return text.replace(regex, "<strong>$1</strong>");
+    }


The function highlightAllMatches is defined but never used in the code. Consider removing it if it's not needed, or implement its usage if it was intended to be part of the functionality.

Suggested change

function highlightAllMatches(text, search) {

const regex = new RegExp(`(${search})`, "gi");

return text.replace(regex, "<strong>$1</strong>");

}

src/sunpy_sphinx_theme/__init__.py

slowe and others added 3 commits June 2, 2025 14:54

Add enhanced search

0e4f98a

Include search js in sphinx build

ca20a4f

Use /_/ proxy on RTD

2cf5674

Cadair reviewed Jun 2, 2025

View reviewed changes

Cadair added 2 commits June 2, 2025 16:59

Use /_/ proxy on RTD

ad1d0d4

Second attempt at using _ RTD api

eee8538

Fix bug with keyboard navigation on project tabs

fbae5cd

Add some default classes

7379115

slowe added 2 commits June 18, 2025 09:46

Tidy up code

8c4d6f1

A more compact version of the URL building. Remove/define some variables.

Change button style

55c4ce8

Switching to classes `sd-btn sd-bg-primary sd-bg-text-primary` from https://sunpy--287.org.readthedocs.build/projects/sunpy-sphinx-theme/287/web-components.html

Add filter icon

3882c00

Cadair mentioned this pull request Jul 8, 2025

List astropy coordinated affiliated packages on main documentation page astropy/astropy#8221

Closed

Cadair added 2 commits September 2, 2025 12:17

Merge branch 'main' into main

54887fd

Add Python side config

015ad25

[pre-commit.ci] auto fixes from pre-commit.com hooks

6d72876

for more information, see https://pre-commit.ci

Cadair requested a review from nabobalis September 2, 2025 14:54

Cadair added 3 commits September 2, 2025 16:37

Extract search CSS and re-org some files

9c3452f

Fix some search css

33d2bd6

Add an editorconfig file because indentation

163fb45

Cadair and others added 4 commits September 2, 2025 17:14

More dark theme fixes

94a7832

Merge remote-tracking branch 'slowe/main' into better-search

676ec99

Keep the linter happy

cde6c9e

Merge branch 'main' into main

f5eed7f

nabobalis reviewed Sep 2, 2025

View reviewed changes

nabobalis requested a review from Copilot September 2, 2025 22:51

Copilot AI reviewed Sep 2, 2025

View reviewed changes

nabobalis approved these changes Sep 2, 2025

View reviewed changes

Uh oh!

Adds enhanced search #287

Are you sure you want to change the base?

Adds enhanced search #287

Uh oh!

Conversation

slowe commented Jun 2, 2025

Uh oh!

Cadair Jun 2, 2025

Choose a reason for hiding this comment

Uh oh!

Cadair Jun 2, 2025

Choose a reason for hiding this comment

Uh oh!

Cadair Jun 2, 2025

Choose a reason for hiding this comment

Uh oh!

Cadair commented Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

slowe commented Jun 13, 2025

Uh oh!

slowe commented Jun 13, 2025

Uh oh!

slowe commented Jun 13, 2025

Uh oh!

slowe commented Jun 13, 2025

Uh oh!

Cadair commented Jun 18, 2025

Uh oh!

slowe commented Jun 18, 2025

Uh oh!

Cadair commented Jun 18, 2025

Uh oh!

slowe commented Jun 24, 2025

Uh oh!

Cadair commented Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Cadair commented Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nabobalis Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Cadair commented Jun 2, 2025 •

edited

Loading

Cadair commented Sep 2, 2025 •

edited

Loading

Cadair commented Sep 2, 2025 •

edited

Loading